Background and Context
Businesses like banks that provide service have to worry about the problem of 'Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on the improvement of service, keeping in mind these priorities.
Objective
Given a Bank customer, build a neural network-based classifier that can determine whether they will leave or not in the next 6 months.
Data Description
The case study is from an open-source dataset from Kaggle. The dataset contains 10,000 sample points with 14 distinct features such as CustomerId, CreditScore, Geography, Gender, Age, Tenure, Balance, etc.
Data Dictionary
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.model_selection import GridSearchCV
import tensorflow as tf
from sklearn import preprocessing
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier, GradientBoostingClassifier, AdaBoostClassifier, StackingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score, roc_curve, classification_report, confusion_matrix
import matplotlib.pyplot as plt
from tensorflow.keras import optimizers
from sklearn.decomposition import PCA
import seaborn as sns
import keras
import tensorflow as tf
from keras import backend as K
from keras.models import Sequential
from keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import Adam
import pandas_profiling
#importing tensorflow
import tensorflow as tf
print(tf.__version__)
2.6.0
#reading dataset
df = pd.read_csv('../../data/bank.csv')
df.head()
| RowNumber | CustomerId | Surname | CreditScore | Geography | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 15634602 | Hargrave | 619 | France | Female | 42 | 2 | 0.00 | 1 | 1 | 1 | 101348.88 | 1 |
| 1 | 2 | 15647311 | Hill | 608 | Spain | Female | 41 | 1 | 83807.86 | 1 | 0 | 1 | 112542.58 | 0 |
| 2 | 3 | 15619304 | Onio | 502 | France | Female | 42 | 8 | 159660.80 | 3 | 1 | 0 | 113931.57 | 1 |
| 3 | 4 | 15701354 | Boni | 699 | France | Female | 39 | 1 | 0.00 | 2 | 0 | 0 | 93826.63 | 0 |
| 4 | 5 | 15737888 | Mitchell | 850 | Spain | Female | 43 | 2 | 125510.82 | 1 | 1 | 1 | 79084.10 | 0 |
wholeReport = pandas_profiling.ProfileReport(df).to_file("wholeReport.html")
pandas_profiling.ProfileReport(df)
RowNumber: Not useful, will be dropped
CustomerId: Not useful, will be dropped
Surname: Not useful, will be dropped
Age: Distribution is relatively skewed with a measure of 1.01. Avg. is 38.9 years, and median is 37 years. Only about 0% missing, inner quartile range is about 12 years. CV is about 26%.
CreditScore: Distribution is slightly left skewed with a measure of -.07. Avg. is 652, and median is 650 years. 0% missing, inner quartile range is about 134 points. CV is about 14.9%.
Geography: France is the most frequent geography, with Germany and Spain close in terms of count. Will OHE.
Gender: 55% male, 45% female. No missing value issue, will OHE.
Tenure: Dstribution isn't skewed. Avg. is 5.0 years, and median is 5 years. About 4% take a value of 0, inner quartile range is about 4 years. CV is about 57%.
Balance: Distribution is relatively skewed with a measure of -.14, though this is due to the large number of 0 values. Avg. is 76485, and median is 97198. About 36% are 0, inner quartile range is about 127644. CV is about 82%.
NumOfProducts: Most customers have either 1 or 2 products. No issues with 0s or missing values.
HasCrCard: 70% have a card while 30% don't. No issues with 0s or missing values.
IsActiveMember: 51% have a card while 49% don't. No issues with 0s or missing values.
EstimatedSalary: Distribution is relatively flat. Avg. is 100k, and median is 100k. No missing value issue, inner quartile range is about 100k. CV is about 57%.
Exited: Target variable. 20% conversion.
ohe_list = [['Geography'],['Gender']]
for att in ohe_list:
df = pd.concat([df,pd.get_dummies(df[att].astype(str), drop_first=True)],axis=1)
df.drop(columns=['RowNumber', 'CustomerId', 'Surname', 'Geography', 'Gender'], inplace = True)
df['churn'] = df['Exited']
df.drop(columns=['Exited'], inplace = True)
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 12 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 CreditScore 10000 non-null int64 1 Age 10000 non-null int64 2 Tenure 10000 non-null int64 3 Balance 10000 non-null float64 4 NumOfProducts 10000 non-null int64 5 HasCrCard 10000 non-null int64 6 IsActiveMember 10000 non-null int64 7 EstimatedSalary 10000 non-null float64 8 Geography_Germany 10000 non-null uint8 9 Geography_Spain 10000 non-null uint8 10 Gender_Male 10000 non-null uint8 11 churn 10000 non-null int64 dtypes: float64(2), int64(7), uint8(3) memory usage: 732.5 KB
#Number of distinct categories or classes i.e., Fraudulent and Genuine
df['churn'].nunique()
2
#checking the percentage of each class in the dataset
(df.churn.value_counts())/(df.churn.count())
0 0.7963 1 0.2037 Name: churn, dtype: float64
print("*********Losses due to fraud:************\n")
print("Total amount lost to churn")
print(df.Balance[df.churn == 1].sum())
print("Mean amount per churn transaction")
print(df.Balance[df.churn == 1].mean())
print("Compare to normal transactions:")
print("Total amount from non-exited")
print(df.Balance[df.churn == 0].sum())
print("Mean amount per non-exited")
print(df.Balance[df.churn == 0].mean())
*********Losses due to fraud:************ Total amount lost to churn 185588094.63 Mean amount per churn transaction 91108.53933726063 Compare to normal transactions: Total amount from non-exited 579270798.25 Mean amount per non-exited 72745.29677885193
#visual representation of instances per class
df.churn.value_counts().plot.bar()
<AxesSubplot:>
#PCA is performed for visualization only
pca= PCA(n_components=2)
churn_2d= pd.DataFrame(data = pca.fit_transform(df))
churn_2d= pd.concat([churn_2d, df['churn']], axis=1)
churn_2d.columns= ['x', 'y', 'churn']
sns.lmplot(x='x', y='y', data=churn_2d, fit_reg=False, hue='churn')
<seaborn.axisgrid.FacetGrid at 0x29654584610>
#Histrogram for feature Time
f, (ax1, ax2) = plt.subplots(2, 1, sharex=True, figsize=(12,4))
ax1.hist(df["CreditScore"][df["churn"] == 1], bins = 50)
ax1.set_title('Exited')
ax2.hist(df["CreditScore"][df["churn"] == 0], bins = 50)
ax2.set_title('Non-exited')
plt.xlabel('Credit Score')
plt.ylabel('Number of Accounts')
plt.show()
Checking correlation between features and the likelihood of the transaction to be fraud on the unbalanced dataset
f, ax1 = plt.subplots(figsize=(24,10))
corr = df.corr()
sns.heatmap(corr, cmap='coolwarm_r', annot_kws={'size':20}, ax=ax1)
ax1.set_title("Correlation Matrix", fontsize=14)
Text(0.5, 1.0, 'Correlation Matrix')
X_data = df.iloc[:,0:11]
y_data = df.iloc[:, -1]
print(y_data.shape)
print(X_data.shape)
(10000,) (10000, 11)
X_data
| CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Geography_Germany | Geography_Spain | Gender_Male | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 619 | 42 | 2 | 0.00 | 1 | 1 | 1 | 101348.88 | 0 | 0 | 0 |
| 1 | 608 | 41 | 1 | 83807.86 | 1 | 0 | 1 | 112542.58 | 0 | 1 | 0 |
| 2 | 502 | 42 | 8 | 159660.80 | 3 | 1 | 0 | 113931.57 | 0 | 0 | 0 |
| 3 | 699 | 39 | 1 | 0.00 | 2 | 0 | 0 | 93826.63 | 0 | 0 | 0 |
| 4 | 850 | 43 | 2 | 125510.82 | 1 | 1 | 1 | 79084.10 | 0 | 1 | 0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 9995 | 771 | 39 | 5 | 0.00 | 2 | 1 | 0 | 96270.64 | 0 | 0 | 1 |
| 9996 | 516 | 35 | 10 | 57369.61 | 1 | 1 | 1 | 101699.77 | 0 | 0 | 1 |
| 9997 | 709 | 36 | 7 | 0.00 | 1 | 0 | 1 | 42085.58 | 0 | 0 | 0 |
| 9998 | 772 | 42 | 3 | 75075.31 | 2 | 1 | 0 | 92888.52 | 1 | 0 | 1 |
| 9999 | 792 | 28 | 4 | 130142.79 | 1 | 1 | 0 | 38190.78 | 0 | 0 | 0 |
10000 rows × 11 columns
y_data
0 1
1 0
2 1
3 0
4 0
..
9995 0
9996 0
9997 1
9998 1
9999 0
Name: churn, Length: 10000, dtype: int64
X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, test_size = 0.2, random_state = 7)
from imblearn.over_sampling import SMOTE
sm = SMOTE(sampling_strategy = 1 ,k_neighbors = 5, random_state=1) #Synthetic Minority Over Sampling Technique
X_train_over, y_train_over = sm.fit_resample(X_train, y_train)
print(X_train_over.shape)
print(X_test.shape)
print(y_train_over.shape)
print(y_test.shape)
print()
print("After UpSampling, counts of label 'Yes': {}".format(sum(y_train_over==1)))
print("After UpSampling, counts of label 'No': {} \n".format(sum(y_train_over==0)))
print('After UpSampling, the shape of train_X: {}'.format(X_train_over.shape))
print('After UpSampling, the shape of train_y: {} \n'.format(y_train_over.shape))
(12748, 11) (2000, 11) (12748,) (2000,) After UpSampling, counts of label 'Yes': 6374 After UpSampling, counts of label 'No': 6374 After UpSampling, the shape of train_X: (12748, 11) After UpSampling, the shape of train_y: (12748,)
## Function to calculate different metric scores of the model - Accuracy, Recall and Precision
def get_metrics_score(model,flag=True):
'''
model : classifier to predict values of X
'''
# defining an empty list to store train and test results
score_list=[]
pred_train = model.predict(X_train)
pred_test = model.predict(X_test)
train_acc = model.score(X_train,y_train)
test_acc = model.score(X_test,y_test)
train_recall = metrics.recall_score(y_train,pred_train)
test_recall = metrics.recall_score(y_test,pred_test)
train_precision = metrics.precision_score(y_train,pred_train)
test_precision = metrics.precision_score(y_test,pred_test)
train_rocAuc = roc_auc_score(y_train, pred_train)
test_rocAuc = roc_auc_score(y_test, pred_test)
score_list.extend((train_acc,test_acc,train_recall,test_recall,train_precision,test_precision, train_rocAuc, test_rocAuc))
# If the flag is set to True then only the following print statements will be dispayed. The default value is set to True.
if flag == True:
print("Accuracy on training set : ",model.score(X_train,y_train))
print("Accuracy on test set : ",model.score(X_test,y_test))
print("Recall on training set : ",metrics.recall_score(y_train,pred_train))
print("Recall on test set : ",metrics.recall_score(y_test,pred_test))
print("Precision on training set : ",metrics.precision_score(y_train,pred_train))
print("Precision on test set : ",metrics.precision_score(y_test,pred_test))
print('ROC-AUC Score on train data:',metrics.roc_auc_score(y_train, pred_train))
print('ROC-AUC Score on test data:',metrics.roc_auc_score(y_test, pred_test))
return score_list # returning the list with train and test scores
### Function to make confusion matrix
def make_confusion_matrix(model,y_actual,labels=[1, 0]):
'''
model : classifier to predict values of X
y_actual : ground truth
'''
y_predict = model.predict(X_test)
cm=metrics.confusion_matrix( y_actual, y_predict, labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["Actual - No","Actual - Yes"]],
columns = [i for i in ['Predicted - No','Predicted - Yes']])
group_counts = ["{0:0.0f}".format(value) for value in
cm.flatten()]
group_percentages = ["{0:.2%}".format(value) for value in
cm.flatten()/np.sum(cm)]
labels = [f"{v1}\n{v2}" for v1, v2 in
zip(group_counts,group_percentages)]
labels = np.asarray(labels).reshape(2,2)
plt.figure(figsize = (10,7))
sns.heatmap(df_cm, annot=labels,fmt='')
plt.ylabel('True label')
plt.xlabel('Predicted label')
# Choose the type of classifier.
estimator = DecisionTreeClassifier(random_state=1)
# Grid of parameters to choose from
parameters = {'max_depth': np.arange(1,10),
'min_samples_leaf': [1, 2, 5, 7, 10,15,20],
'max_leaf_nodes' : [5, 10,15,20,25,30],
'min_impurity_decrease': [0.0001,0.001,0.01,0.1]
}
# Type of scoring used to compare parameter combinations
acc_scorer = metrics.make_scorer(metrics.recall_score)
# Run the grid search
grid_obj = GridSearchCV(estimator, parameters, scoring=acc_scorer,cv=5)
grid_obj = grid_obj.fit(X_train_over, y_train_over)
# Set the clf to the best combination of parameters
estimator = grid_obj.best_estimator_
# Fit the best algorithm to the data.
estimator.fit(X_train_over, y_train_over)
#Calculating different metrics
get_metrics_score(estimator)
#Creating confusion matrix
make_confusion_matrix(estimator,y_test)
Accuracy on training set : 0.695875 Accuracy on test set : 0.6835 Recall on training set : 0.7300123001230012 Recall on test set : 0.7396593673965937 Precision on training set : 0.3731530965105313 Precision on test set : 0.36626506024096384 ROC-AUC Score on train data: 0.7085894572469413 ROC-AUC Score on test data: 0.7043167825025761
Recall to be maximized, greater the Recall higher the chances of minimizing false Negative. Hence, the focus should be on increasing Recall or minimizing the false Negative or in other words identifying the True Positive(i.e. Class 1) so that the Company can identify the churn accounts.#Standardizing the Amount column (All other 'V' columns are already scaled as they've undergone PCA transformation).
from sklearn.preprocessing import StandardScaler
features = list(X_data.columns)
print(features)
# Separating out the features
x = df.loc[:, features].values
# Separating out the target
y = df.loc[:,['churn']].values
# Standardizing the features
x = StandardScaler().fit_transform(x)
X_data = pd.DataFrame(x, columns = features)
y_Data = pd.DataFrame(y, columns = ['churn'])
#Xt_data[list(X_data.columns)].values = StandardScaler().fit_transform(X_data[list(X_data.columns)].values)
#Xt_data= Xt_data.drop(['Amount'],axis=1)
['CreditScore', 'Age', 'Tenure', 'Balance', 'NumOfProducts', 'HasCrCard', 'IsActiveMember', 'EstimatedSalary', 'Geography_Germany', 'Geography_Spain', 'Gender_Male']
X_data
| CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Geography_Germany | Geography_Spain | Gender_Male | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -0.326221 | 0.293517 | -1.041760 | -1.225848 | -0.911583 | 0.646092 | 0.970243 | 0.021886 | -0.578736 | -0.573809 | -1.095988 |
| 1 | -0.440036 | 0.198164 | -1.387538 | 0.117350 | -0.911583 | -1.547768 | 0.970243 | 0.216534 | -0.578736 | 1.742740 | -1.095988 |
| 2 | -1.536794 | 0.293517 | 1.032908 | 1.333053 | 2.527057 | 0.646092 | -1.030670 | 0.240687 | -0.578736 | -0.573809 | -1.095988 |
| 3 | 0.501521 | 0.007457 | -1.387538 | -1.225848 | 0.807737 | -1.547768 | -1.030670 | -0.108918 | -0.578736 | -0.573809 | -1.095988 |
| 4 | 2.063884 | 0.388871 | -1.041760 | 0.785728 | -0.911583 | 0.646092 | 0.970243 | -0.365276 | -0.578736 | 1.742740 | -1.095988 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 9995 | 1.246488 | 0.007457 | -0.004426 | -1.225848 | 0.807737 | 0.646092 | -1.030670 | -0.066419 | -0.578736 | -0.573809 | 0.912419 |
| 9996 | -1.391939 | -0.373958 | 1.724464 | -0.306379 | -0.911583 | 0.646092 | 0.970243 | 0.027988 | -0.578736 | -0.573809 | 0.912419 |
| 9997 | 0.604988 | -0.278604 | 0.687130 | -1.225848 | -0.911583 | -1.547768 | 0.970243 | -1.008643 | -0.578736 | -0.573809 | -1.095988 |
| 9998 | 1.256835 | 0.293517 | -0.695982 | -0.022608 | 0.807737 | 0.646092 | -1.030670 | -0.125231 | 1.727904 | -0.573809 | 0.912419 |
| 9999 | 1.463771 | -1.041433 | -0.350204 | 0.859965 | -0.911583 | 0.646092 | -1.030670 | -1.076370 | -0.578736 | -0.573809 | -1.095988 |
10000 rows × 11 columns
X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, test_size = 0.2, random_state = 7)
sm = SMOTE(sampling_strategy = 1 ,k_neighbors = 5, random_state=1) #Synthetic Minority Over Sampling Technique
X_train_over, y_train_over = sm.fit_resample(X_train, y_train)
print()
print("After UpSampling, counts of label 'Yes': {}".format(sum(y_train_over==1)))
print("After UpSampling, counts of label 'No': {} \n".format(sum(y_train_over==0)))
print('After UpSampling, the shape of train_X: {}'.format(X_train_over.shape))
print('After UpSampling, the shape of train_y: {} \n'.format(y_train_over.shape))
After UpSampling, counts of label 'Yes': 6374 After UpSampling, counts of label 'No': 6374 After UpSampling, the shape of train_X: (12748, 11) After UpSampling, the shape of train_y: (12748,)
Model-1
Dropout
Dropout is a regularization technique for neural network models proposed by Srivastava, et al. in their 2014 paper Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Dropout is a technique where randomly selected neurons are ignored during training. They are “dropped-out” randomly.
Keras model object can be created with Sequential class
At the outset, the model is empty per se. It is completed by adding additional layers and compilation
#initialize the model
model = Sequential()
Keras layers can be added to the model
Adding layers are like stacking lego blocks one by one
It should be noted that as this is a classification problem, sigmoid layer (softmax for multi-class problems) should be added
# This adds the input layer (by specifying input dimension) AND the first hidden layer (units)
model.add(Dense(units=6, input_dim = 11,activation='relu')) # input of 29 columns as shown above
# hidden layer
model.add(Dense(units=6,activation='relu'))
#Adding Dropout to prevent overfitting
model.add(Dropout(0.5))
model.add(Dense(24,activation='relu'))
model.add(Dense(24,activation='relu'))
# Adding the output layer
# Notice that we do not need to specify input dim.
# we have an output of 1 node, which is the the desired dimensions of our output (fraud or not)
# We use the sigmoid because we want probability outcomes
model.add(Dense(1,activation='sigmoid')) # binary classification fraudulent or not
Keras model should be "compiled" prior to training
Types of loss (function) and optimizer should be designated
def recall_m(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
return recall
# Create optimizer with default learning rate
# Compile the model
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=[recall_m])
model.summary()
Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense (Dense) (None, 6) 72 _________________________________________________________________ dense_1 (Dense) (None, 6) 42 _________________________________________________________________ dropout (Dropout) (None, 6) 0 _________________________________________________________________ dense_2 (Dense) (None, 24) 168 _________________________________________________________________ dense_3 (Dense) (None, 24) 600 _________________________________________________________________ dense_4 (Dense) (None, 1) 25 ================================================================= Total params: 907 Trainable params: 907 Non-trainable params: 0 _________________________________________________________________
Training the model
#fitting the model
history=model.fit(X_train_over,y_train_over,batch_size=10,epochs=100,validation_split=0.2)
Epoch 1/100 1020/1020 [==============================] - 2s 2ms/step - loss: 0.6500 - recall_m: 0.0537 - val_loss: 0.8545 - val_recall_m: 0.3369 Epoch 2/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5903 - recall_m: 0.3466 - val_loss: 0.8166 - val_recall_m: 0.5090 Epoch 3/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5649 - recall_m: 0.4272 - val_loss: 0.7585 - val_recall_m: 0.5212 Epoch 4/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5489 - recall_m: 0.4780 - val_loss: 0.7308 - val_recall_m: 0.6588 Epoch 5/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5352 - recall_m: 0.6406 - val_loss: 0.6986 - val_recall_m: 0.6843 Epoch 6/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5261 - recall_m: 0.6607 - val_loss: 0.6267 - val_recall_m: 0.7188 Epoch 7/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5134 - recall_m: 0.7103 - val_loss: 0.6120 - val_recall_m: 0.7353 Epoch 8/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5122 - recall_m: 0.7269 - val_loss: 0.6571 - val_recall_m: 0.6969 Epoch 9/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5021 - recall_m: 0.7139 - val_loss: 0.6223 - val_recall_m: 0.6886 Epoch 10/100 1020/1020 [==============================] - 1s 918us/step - loss: 0.5043 - recall_m: 0.7112 - val_loss: 0.5854 - val_recall_m: 0.7278 Epoch 11/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4992 - recall_m: 0.7083 - val_loss: 0.6582 - val_recall_m: 0.6855 Epoch 12/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4981 - recall_m: 0.7047 - val_loss: 0.6610 - val_recall_m: 0.6804 Epoch 13/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4960 - recall_m: 0.7017 - val_loss: 0.6398 - val_recall_m: 0.6882 Epoch 14/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4943 - recall_m: 0.7036 - val_loss: 0.5421 - val_recall_m: 0.7502 Epoch 15/100 1020/1020 [==============================] - 1s 1000us/step - loss: 0.4950 - recall_m: 0.7165 - val_loss: 0.6001 - val_recall_m: 0.6835 Epoch 16/100 1020/1020 [==============================] - 1s 977us/step - loss: 0.4941 - recall_m: 0.7015 - val_loss: 0.5985 - val_recall_m: 0.7149 Epoch 17/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4916 - recall_m: 0.7026 - val_loss: 0.5795 - val_recall_m: 0.7063 Epoch 18/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4932 - recall_m: 0.7024 - val_loss: 0.6317 - val_recall_m: 0.5678 Epoch 19/100 1020/1020 [==============================] - 1s 907us/step - loss: 0.4894 - recall_m: 0.6901 - val_loss: 0.5983 - val_recall_m: 0.6682 Epoch 20/100 1020/1020 [==============================] - 1s 927us/step - loss: 0.4870 - recall_m: 0.6891 - val_loss: 0.5648 - val_recall_m: 0.6886 Epoch 21/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4848 - recall_m: 0.6704 - val_loss: 0.5056 - val_recall_m: 0.7537 Epoch 22/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4849 - recall_m: 0.6574 - val_loss: 0.4648 - val_recall_m: 0.7941 Epoch 23/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4766 - recall_m: 0.6944 - val_loss: 0.5064 - val_recall_m: 0.7439 Epoch 24/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4787 - recall_m: 0.6617 - val_loss: 0.5891 - val_recall_m: 0.6749 Epoch 25/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4793 - recall_m: 0.6682 - val_loss: 0.5298 - val_recall_m: 0.7004 Epoch 26/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4784 - recall_m: 0.6662 - val_loss: 0.4162 - val_recall_m: 0.7988 Epoch 27/100 1020/1020 [==============================] - 2s 2ms/step - loss: 0.4786 - recall_m: 0.6587 - val_loss: 0.4983 - val_recall_m: 0.7510 Epoch 28/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4786 - recall_m: 0.6458 - val_loss: 0.4629 - val_recall_m: 0.7839 Epoch 29/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4773 - recall_m: 0.6610 - val_loss: 0.4870 - val_recall_m: 0.7871 Epoch 30/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4756 - recall_m: 0.6605 - val_loss: 0.5115 - val_recall_m: 0.7502 Epoch 31/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4745 - recall_m: 0.6665 - val_loss: 0.3957 - val_recall_m: 0.8671 Epoch 32/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4751 - recall_m: 0.6823 - val_loss: 0.4672 - val_recall_m: 0.7373 Epoch 33/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4700 - recall_m: 0.6487 - val_loss: 0.4168 - val_recall_m: 0.8494 Epoch 34/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4710 - recall_m: 0.6797 - val_loss: 0.5008 - val_recall_m: 0.7608 Epoch 35/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4729 - recall_m: 0.6650 - val_loss: 0.4385 - val_recall_m: 0.8110 Epoch 36/100 1020/1020 [==============================] - 1s 792us/step - loss: 0.4711 - recall_m: 0.6530 - val_loss: 0.4585 - val_recall_m: 0.8184 Epoch 37/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4736 - recall_m: 0.6629 - val_loss: 0.4544 - val_recall_m: 0.7914 Epoch 38/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4751 - recall_m: 0.6474 - val_loss: 0.3851 - val_recall_m: 0.8929 Epoch 39/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4691 - recall_m: 0.6607 - val_loss: 0.3316 - val_recall_m: 0.9067 Epoch 40/100 1020/1020 [==============================] - 1s 953us/step - loss: 0.4685 - recall_m: 0.6666 - val_loss: 0.4302 - val_recall_m: 0.8184 Epoch 41/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4711 - recall_m: 0.6648 - val_loss: 0.4132 - val_recall_m: 0.9090 Epoch 42/100 1020/1020 [==============================] - 1s 926us/step - loss: 0.4703 - recall_m: 0.6692 - val_loss: 0.4075 - val_recall_m: 0.9306 Epoch 43/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4714 - recall_m: 0.6674 - val_loss: 0.4168 - val_recall_m: 0.8631 Epoch 44/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4685 - recall_m: 0.6581 - val_loss: 0.3603 - val_recall_m: 0.9133 Epoch 45/100 1020/1020 [==============================] - 1s 822us/step - loss: 0.4705 - recall_m: 0.6518 - val_loss: 0.3817 - val_recall_m: 0.8820 Epoch 46/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4671 - recall_m: 0.6341 - val_loss: 0.4163 - val_recall_m: 0.8651 Epoch 47/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4683 - recall_m: 0.6503 - val_loss: 0.3397 - val_recall_m: 0.9122 Epoch 48/100 1020/1020 [==============================] - 1s 979us/step - loss: 0.4634 - recall_m: 0.6660 - val_loss: 0.3019 - val_recall_m: 0.9290 Epoch 49/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4633 - recall_m: 0.6561 - val_loss: 0.3363 - val_recall_m: 0.9145 Epoch 50/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4655 - recall_m: 0.6544 - val_loss: 0.3132 - val_recall_m: 0.9216 Epoch 51/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4644 - recall_m: 0.6382 - val_loss: 0.3028 - val_recall_m: 0.9082 Epoch 52/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4634 - recall_m: 0.6274 - val_loss: 0.2994 - val_recall_m: 0.9220 Epoch 53/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4642 - recall_m: 0.6059 - val_loss: 0.3028 - val_recall_m: 0.9431 Epoch 54/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4638 - recall_m: 0.6226 - val_loss: 0.2468 - val_recall_m: 0.9733 Epoch 55/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4610 - recall_m: 0.6491 - val_loss: 0.2727 - val_recall_m: 0.9537 Epoch 56/100 1020/1020 [==============================] - 1s 994us/step - loss: 0.4617 - recall_m: 0.6166 - val_loss: 0.2233 - val_recall_m: 0.9827 Epoch 57/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4587 - recall_m: 0.5730 - val_loss: 0.2393 - val_recall_m: 0.9757 Epoch 58/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4589 - recall_m: 0.5961 - val_loss: 0.2747 - val_recall_m: 0.9557 Epoch 59/100 1020/1020 [==============================] - 1s 744us/step - loss: 0.4565 - recall_m: 0.6042 - val_loss: 0.2620 - val_recall_m: 0.9757 Epoch 60/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4604 - recall_m: 0.5893 - val_loss: 0.2608 - val_recall_m: 0.9655 Epoch 61/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4561 - recall_m: 0.5899 - val_loss: 0.2796 - val_recall_m: 0.9404 Epoch 62/100 1020/1020 [==============================] - 1s 837us/step - loss: 0.4546 - recall_m: 0.5857 - val_loss: 0.2538 - val_recall_m: 0.9588 Epoch 63/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4536 - recall_m: 0.5795 - val_loss: 0.1776 - val_recall_m: 0.9851 Epoch 64/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4560 - recall_m: 0.5786 - val_loss: 0.1834 - val_recall_m: 0.9737 Epoch 65/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4542 - recall_m: 0.5828 - val_loss: 0.1945 - val_recall_m: 0.9820 Epoch 66/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4548 - recall_m: 0.5797 - val_loss: 0.1515 - val_recall_m: 0.9878 Epoch 67/100 1020/1020 [==============================] - 1s 995us/step - loss: 0.4537 - recall_m: 0.5934 - val_loss: 0.1461 - val_recall_m: 0.9894 Epoch 68/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4517 - recall_m: 0.5984 - val_loss: 0.1527 - val_recall_m: 0.9878 Epoch 69/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4517 - recall_m: 0.6022 - val_loss: 0.1692 - val_recall_m: 0.9757 Epoch 70/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4494 - recall_m: 0.5871 - val_loss: 0.1457 - val_recall_m: 0.9843 Epoch 71/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4497 - recall_m: 0.6000 - val_loss: 0.1425 - val_recall_m: 0.9855 Epoch 72/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4501 - recall_m: 0.6065 - val_loss: 0.1664 - val_recall_m: 0.9812 Epoch 73/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4474 - recall_m: 0.6085 - val_loss: 0.1632 - val_recall_m: 0.9902 Epoch 74/100 1020/1020 [==============================] - 1s 912us/step - loss: 0.4489 - recall_m: 0.6065 - val_loss: 0.1265 - val_recall_m: 0.9929 Epoch 75/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4488 - recall_m: 0.6309 - val_loss: 0.1847 - val_recall_m: 0.9855 Epoch 76/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4464 - recall_m: 0.6178 - val_loss: 0.1679 - val_recall_m: 0.9816 Epoch 77/100 1020/1020 [==============================] - 1s 987us/step - loss: 0.4495 - recall_m: 0.6084 - val_loss: 0.1531 - val_recall_m: 0.9890 Epoch 78/100 1020/1020 [==============================] - 1s 923us/step - loss: 0.4487 - recall_m: 0.6044 - val_loss: 0.1794 - val_recall_m: 0.9886 Epoch 79/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4495 - recall_m: 0.6160 - val_loss: 0.1407 - val_recall_m: 0.9949 Epoch 80/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4453 - recall_m: 0.6170 - val_loss: 0.1507 - val_recall_m: 0.9937 Epoch 81/100 1020/1020 [==============================] - 1s 930us/step - loss: 0.4482 - recall_m: 0.6087 - val_loss: 0.1775 - val_recall_m: 0.9894 Epoch 82/100 1020/1020 [==============================] - 1s 845us/step - loss: 0.4443 - recall_m: 0.6149 - val_loss: 0.1932 - val_recall_m: 0.9843 Epoch 83/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4481 - recall_m: 0.6064 - val_loss: 0.1815 - val_recall_m: 0.9875 Epoch 84/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4487 - recall_m: 0.6119 - val_loss: 0.1873 - val_recall_m: 0.9859 Epoch 85/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4453 - recall_m: 0.6199 - val_loss: 0.1548 - val_recall_m: 0.9929 Epoch 86/100 1020/1020 [==============================] - 1s 855us/step - loss: 0.4481 - recall_m: 0.6368 - val_loss: 0.2170 - val_recall_m: 0.9784 Epoch 87/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4465 - recall_m: 0.6335 - val_loss: 0.1646 - val_recall_m: 0.9933 Epoch 88/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4457 - recall_m: 0.6303 - val_loss: 0.2369 - val_recall_m: 0.9765 Epoch 89/100 1020/1020 [==============================] - 1s 960us/step - loss: 0.4424 - recall_m: 0.6382 - val_loss: 0.1680 - val_recall_m: 0.9875 Epoch 90/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4446 - recall_m: 0.6209 - val_loss: 0.1739 - val_recall_m: 0.9835 Epoch 91/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4457 - recall_m: 0.6282 - val_loss: 0.1639 - val_recall_m: 0.9875 Epoch 92/100 1020/1020 [==============================] - 1s 817us/step - loss: 0.4436 - recall_m: 0.6327 - val_loss: 0.1600 - val_recall_m: 0.9929 Epoch 93/100 1020/1020 [==============================] - 1s 897us/step - loss: 0.4437 - recall_m: 0.6116 - val_loss: 0.1786 - val_recall_m: 0.9867 Epoch 94/100 1020/1020 [==============================] - 1s 872us/step - loss: 0.4451 - recall_m: 0.6362 - val_loss: 0.1424 - val_recall_m: 0.9937 Epoch 95/100 1020/1020 [==============================] - 1s 848us/step - loss: 0.4424 - recall_m: 0.6188 - val_loss: 0.1501 - val_recall_m: 0.9925 Epoch 96/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4409 - recall_m: 0.6284 - val_loss: 0.1761 - val_recall_m: 0.9835 Epoch 97/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4424 - recall_m: 0.6298 - val_loss: 0.1322 - val_recall_m: 0.9965 Epoch 98/100 1020/1020 [==============================] - 1s 959us/step - loss: 0.4438 - recall_m: 0.6294 - val_loss: 0.1558 - val_recall_m: 0.9831 Epoch 99/100 1020/1020 [==============================] - 1s 867us/step - loss: 0.4455 - recall_m: 0.6342 - val_loss: 0.1769 - val_recall_m: 0.9859 Epoch 100/100 1020/1020 [==============================] - 1s 813us/step - loss: 0.4426 - recall_m: 0.6285 - val_loss: 0.1843 - val_recall_m: 0.9914
Plotting the train and test loss
# Capturing learning history per epoch
hist = pd.DataFrame(history.history)
hist['epoch'] = history.epoch
# Plotting accuracy at different epochs
plt.plot(hist['loss'])
plt.plot(hist['val_loss'])
plt.legend(("train" , "valid") , loc =0)
<matplotlib.legend.Legend at 0x296269e3040>
Keras model can be evaluated with evaluate() function
Evaluation results are contained in a list
score = model.evaluate(X_test, y_test)
63/63 [==============================] - 0s 2ms/step - loss: 0.9199 - recall_m: 0.9792
print(score)
[0.9199440479278564, 0.979205846786499]
def make_confusion_matrix(cf,
group_names=None,
categories='auto',
count=True,
percent=True,
cbar=True,
xyticks=True,
xyplotlabels=True,
sum_stats=True,
figsize=None,
cmap='Blues',
title=None):
'''
This function will make a pretty plot of an sklearn Confusion Matrix cm using a Seaborn heatmap visualization.
Arguments
'''
# CODE TO GENERATE TEXT INSIDE EACH SQUARE
blanks = ['' for i in range(cf.size)]
if group_names and len(group_names)==cf.size:
group_labels = ["{}\n".format(value) for value in group_names]
else:
group_labels = blanks
if count:
group_counts = ["{0:0.0f}\n".format(value) for value in cf.flatten()]
else:
group_counts = blanks
if percent:
group_percentages = ["{0:.2%}".format(value) for value in cf.flatten()/np.sum(cf)]
else:
group_percentages = blanks
box_labels = [f"{v1}{v2}{v3}".strip() for v1, v2, v3 in zip(group_labels,group_counts,group_percentages)]
box_labels = np.asarray(box_labels).reshape(cf.shape[0],cf.shape[1])
# CODE TO GENERATE SUMMARY STATISTICS & TEXT FOR SUMMARY STATS
if sum_stats:
#Accuracy is sum of diagonal divided by total observations
accuracy = np.trace(cf) / float(np.sum(cf))
#if it is a binary confusion matrix, show some more stats
if len(cf)==2:
#Metrics for Binary Confusion Matrices
precision = cf[1,1] / sum(cf[:,1])
recall = cf[1,1] / sum(cf[1,:])
f1_score = 2*precision*recall / (precision + recall)
stats_text = "\n\nAccuracy={:0.3f}\nPrecision={:0.3f}\nRecall={:0.3f}\nF1 Score={:0.3f}".format(
accuracy,precision,recall,f1_score)
else:
stats_text = "\n\nAccuracy={:0.3f}".format(accuracy)
else:
stats_text = ""
# SET FIGURE PARAMETERS ACCORDING TO OTHER ARGUMENTS
if figsize==None:
#Get default figure size if not set
figsize = plt.rcParams.get('figure.figsize')
if xyticks==False:
#Do not show categories if xyticks is False
categories=False
# MAKE THE HEATMAP VISUALIZATION
plt.figure(figsize=figsize)
sns.heatmap(cf,annot=box_labels,fmt="",cmap=cmap,cbar=cbar,xticklabels=categories,yticklabels=categories)
if xyplotlabels:
plt.ylabel('True label')
plt.xlabel('Predicted label' + stats_text)
else:
plt.xlabel(stats_text)
if title:
plt.title(title)
## Confusion Matrix on unsee test set
import seaborn as sn
y_pred1 = model.predict(X_test)
for i in range(len(y_test)):
if y_pred1[i]>0.5:
y_pred1[i]=1
else:
y_pred1[i]=0
cm2=confusion_matrix(y_test, y_pred1)
labels = ['True Negative','False Positive','False Negative','True Positive']
categories = [ 'Non_Exited','Exited']
make_confusion_matrix(cm2,
group_names=labels,
categories=categories,
cmap='Blues')
#initialize the model
model = Sequential()
# This adds the input layer (by specifying input dimension) AND the first hidden layer (units)
model.add(Dense(units=6, input_dim = 11,activation='relu')) # input of 29 columns as shown above
# hidden layer
model.add(Dense(units=6,activation='relu'))
#Adding Dropout to prevent overfitting
model.add(Dropout(0.5))
model.add(Dense(24,activation='relu'))
model.add(Dense(24,activation='relu'))
# Adding the output layer
# Notice that we do not need to specify input dim.
# we have an output of 1 node, which is the the desired dimensions of our output (fraud or not)
# We use the sigmoid because we want probability outcomes
model.add(Dense(1,activation='sigmoid')) # binary classification fraudulent or not
def recall_m(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
return recall
# Create optimizer with default learning rate
# Compile the model
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=[tf.keras.metrics.AUC()])
model.summary()
Model: "sequential_1" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_5 (Dense) (None, 6) 72 _________________________________________________________________ dense_6 (Dense) (None, 6) 42 _________________________________________________________________ dropout_1 (Dropout) (None, 6) 0 _________________________________________________________________ dense_7 (Dense) (None, 24) 168 _________________________________________________________________ dense_8 (Dense) (None, 24) 600 _________________________________________________________________ dense_9 (Dense) (None, 1) 25 ================================================================= Total params: 907 Trainable params: 907 Non-trainable params: 0 _________________________________________________________________
#fitting the model
history=model.fit(X_train_over,y_train_over,batch_size=10,epochs=100,validation_split=0.2)
Epoch 1/100 1020/1020 [==============================] - 2s 1ms/step - loss: 0.6285 - auc: 0.6442 - val_loss: 0.7816 - val_auc: 0.0000e+00 Epoch 2/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5640 - auc: 0.7546 - val_loss: 0.6953 - val_auc: 0.0000e+00 Epoch 3/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5365 - auc: 0.7844 - val_loss: 0.6498 - val_auc: 0.0000e+00 Epoch 4/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5238 - auc: 0.7975 - val_loss: 0.6002 - val_auc: 0.0000e+00 Epoch 5/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.5091 - auc: 0.8110 - val_loss: 0.5654 - val_auc: 0.0000e+00 Epoch 6/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4998 - auc: 0.8179 - val_loss: 0.5661 - val_auc: 0.0000e+00 Epoch 7/100 1020/1020 [==============================] - 1s 975us/step - loss: 0.4898 - auc: 0.8274 - val_loss: 0.5340 - val_auc: 0.0000e+00 Epoch 8/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4893 - auc: 0.8287 - val_loss: 0.5802 - val_auc: 0.0000e+00 Epoch 9/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4875 - auc: 0.8282 - val_loss: 0.4994 - val_auc: 0.0000e+00 Epoch 10/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4832 - auc: 0.8321 - val_loss: 0.4979 - val_auc: 0.0000e+00 Epoch 11/100 1020/1020 [==============================] - 1s 932us/step - loss: 0.4794 - auc: 0.8356 - val_loss: 0.5250 - val_auc: 0.0000e+00 Epoch 12/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4733 - auc: 0.8394 - val_loss: 0.4935 - val_auc: 0.0000e+00 Epoch 13/100 1020/1020 [==============================] - 1s 958us/step - loss: 0.4749 - auc: 0.8394 - val_loss: 0.5095 - val_auc: 0.0000e+00 Epoch 14/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4741 - auc: 0.8397 - val_loss: 0.4405 - val_auc: 0.0000e+00 Epoch 15/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4687 - auc: 0.8445 - val_loss: 0.4186 - val_auc: 0.0000e+00 Epoch 16/100 1020/1020 [==============================] - 1s 926us/step - loss: 0.4663 - auc: 0.8451 - val_loss: 0.4142 - val_auc: 0.0000e+00 Epoch 17/100 1020/1020 [==============================] - 1s 938us/step - loss: 0.4646 - auc: 0.8466 - val_loss: 0.4263 - val_auc: 0.0000e+00 Epoch 18/100 1020/1020 [==============================] - 1s 999us/step - loss: 0.4622 - auc: 0.8481 - val_loss: 0.4118 - val_auc: 0.0000e+00 Epoch 19/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4638 - auc: 0.8474 - val_loss: 0.4146 - val_auc: 0.0000e+00 Epoch 20/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4606 - auc: 0.8498 - val_loss: 0.4526 - val_auc: 0.0000e+00 Epoch 21/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4577 - auc: 0.8517 - val_loss: 0.3558 - val_auc: 0.0000e+00 Epoch 22/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4609 - auc: 0.8498 - val_loss: 0.3946 - val_auc: 0.0000e+00 Epoch 23/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4575 - auc: 0.8525 - val_loss: 0.3952 - val_auc: 0.0000e+00 Epoch 24/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4620 - auc: 0.8495 - val_loss: 0.3761 - val_auc: 0.0000e+00 Epoch 25/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4620 - auc: 0.8484 - val_loss: 0.4445 - val_auc: 0.0000e+00 Epoch 26/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4579 - auc: 0.8523 - val_loss: 0.3863 - val_auc: 0.0000e+00 Epoch 27/100 1020/1020 [==============================] - 1s 988us/step - loss: 0.4592 - auc: 0.8511 - val_loss: 0.3661 - val_auc: 0.0000e+00 Epoch 28/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4606 - auc: 0.8504 - val_loss: 0.3476 - val_auc: 0.0000e+00 Epoch 29/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4600 - auc: 0.8500 - val_loss: 0.3813 - val_auc: 0.0000e+00 Epoch 30/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4582 - auc: 0.8517 - val_loss: 0.4191 - val_auc: 0.0000e+00 Epoch 31/100 1020/1020 [==============================] - 1s 937us/step - loss: 0.4578 - auc: 0.8519 - val_loss: 0.3765 - val_auc: 0.0000e+00 Epoch 32/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4540 - auc: 0.8546 - val_loss: 0.3594 - val_auc: 0.0000e+00 Epoch 33/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4575 - auc: 0.8521 - val_loss: 0.3716 - val_auc: 0.0000e+00 Epoch 34/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4542 - auc: 0.8549 - val_loss: 0.3755 - val_auc: 0.0000e+00 Epoch 35/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4578 - auc: 0.8522 - val_loss: 0.3346 - val_auc: 0.0000e+00 Epoch 36/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4554 - auc: 0.8539 - val_loss: 0.3422 - val_auc: 0.0000e+00 Epoch 37/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4524 - auc: 0.8561 - val_loss: 0.3498 - val_auc: 0.0000e+00 Epoch 38/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4532 - auc: 0.8558 - val_loss: 0.4018 - val_auc: 0.0000e+00 Epoch 39/100 1020/1020 [==============================] - 2s 2ms/step - loss: 0.4513 - auc: 0.8565 - val_loss: 0.3469 - val_auc: 0.0000e+00 Epoch 40/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4529 - auc: 0.8559 - val_loss: 0.3312 - val_auc: 0.0000e+00 Epoch 41/100 1020/1020 [==============================] - 1s 882us/step - loss: 0.4509 - auc: 0.8566 - val_loss: 0.3663 - val_auc: 0.0000e+00 Epoch 42/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4487 - auc: 0.8585 - val_loss: 0.2977 - val_auc: 0.0000e+00 Epoch 43/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4541 - auc: 0.8550 - val_loss: 0.3505 - val_auc: 0.0000e+00 Epoch 44/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4536 - auc: 0.8553 - val_loss: 0.3870 - val_auc: 0.0000e+00 Epoch 45/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4567 - auc: 0.8535 - val_loss: 0.3532 - val_auc: 0.0000e+00 Epoch 46/100 1020/1020 [==============================] - 1s 946us/step - loss: 0.4525 - auc: 0.8562 - val_loss: 0.3704 - val_auc: 0.0000e+00 Epoch 47/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4528 - auc: 0.8559 - val_loss: 0.3535 - val_auc: 0.0000e+00 Epoch 48/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4542 - auc: 0.8549 - val_loss: 0.3607 - val_auc: 0.0000e+00 Epoch 49/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4518 - auc: 0.8567 - val_loss: 0.3337 - val_auc: 0.0000e+00 Epoch 50/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4531 - auc: 0.8555 - val_loss: 0.3709 - val_auc: 0.0000e+00 Epoch 51/100 1020/1020 [==============================] - 1s 837us/step - loss: 0.4508 - auc: 0.8572 - val_loss: 0.3948 - val_auc: 0.0000e+00 Epoch 52/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4513 - auc: 0.8573 - val_loss: 0.3747 - val_auc: 0.0000e+00 Epoch 53/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4532 - auc: 0.8558 - val_loss: 0.3242 - val_auc: 0.0000e+00 Epoch 54/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4516 - auc: 0.8563 - val_loss: 0.3746 - val_auc: 0.0000e+00 Epoch 55/100 1020/1020 [==============================] - 1s 848us/step - loss: 0.4521 - auc: 0.8567 - val_loss: 0.3360 - val_auc: 0.0000e+00 Epoch 56/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4516 - auc: 0.8572 - val_loss: 0.3953 - val_auc: 0.0000e+00 Epoch 57/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4530 - auc: 0.8557 - val_loss: 0.3519 - val_auc: 0.0000e+00 Epoch 58/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4488 - auc: 0.8588 - val_loss: 0.4451 - val_auc: 0.0000e+00 Epoch 59/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4488 - auc: 0.8588 - val_loss: 0.3518 - val_auc: 0.0000e+00 Epoch 60/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4486 - auc: 0.8588 - val_loss: 0.3454 - val_auc: 0.0000e+00 Epoch 61/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4503 - auc: 0.8572 - val_loss: 0.3653 - val_auc: 0.0000e+00 Epoch 62/100 1020/1020 [==============================] - 1s 901us/step - loss: 0.4519 - auc: 0.8569 - val_loss: 0.3689 - val_auc: 0.0000e+00 Epoch 63/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4553 - auc: 0.8549 - val_loss: 0.3367 - val_auc: 0.0000e+00 Epoch 64/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4475 - auc: 0.8596 - val_loss: 0.3525 - val_auc: 0.0000e+00 Epoch 65/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4521 - auc: 0.8562 - val_loss: 0.3352 - val_auc: 0.0000e+00 Epoch 66/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4518 - auc: 0.8564 - val_loss: 0.3161 - val_auc: 0.0000e+00 Epoch 67/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4512 - auc: 0.8573 - val_loss: 0.3636 - val_auc: 0.0000e+00 Epoch 68/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4500 - auc: 0.8580 - val_loss: 0.3159 - val_auc: 0.0000e+00 Epoch 69/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4486 - auc: 0.8589 - val_loss: 0.3458 - val_auc: 0.0000e+00 Epoch 70/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4480 - auc: 0.8594 - val_loss: 0.3323 - val_auc: 0.0000e+00 Epoch 71/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4466 - auc: 0.8595 - val_loss: 0.3548 - val_auc: 0.0000e+00 Epoch 72/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4506 - auc: 0.8571 - val_loss: 0.3713 - val_auc: 0.0000e+00 Epoch 73/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4553 - auc: 0.8543 - val_loss: 0.3596 - val_auc: 0.0000e+00 Epoch 74/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4539 - auc: 0.8557 - val_loss: 0.3285 - val_auc: 0.0000e+00 Epoch 75/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4518 - auc: 0.8565 - val_loss: 0.3390 - val_auc: 0.0000e+00 Epoch 76/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4493 - auc: 0.8590 - val_loss: 0.3459 - val_auc: 0.0000e+00 Epoch 77/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4517 - auc: 0.8570 - val_loss: 0.3660 - val_auc: 0.0000e+00 Epoch 78/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4501 - auc: 0.8580 - val_loss: 0.3849 - val_auc: 0.0000e+00 Epoch 79/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4470 - auc: 0.8595 - val_loss: 0.3570 - val_auc: 0.0000e+00 Epoch 80/100 1020/1020 [==============================] - 1s 982us/step - loss: 0.4518 - auc: 0.8571 - val_loss: 0.3780 - val_auc: 0.0000e+00 Epoch 81/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4520 - auc: 0.8564 - val_loss: 0.4041 - val_auc: 0.0000e+00 Epoch 82/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4502 - auc: 0.8572 - val_loss: 0.3512 - val_auc: 0.0000e+00 Epoch 83/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4452 - auc: 0.8609 - val_loss: 0.3416 - val_auc: 0.0000e+00 Epoch 84/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4504 - auc: 0.8579 - val_loss: 0.3570 - val_auc: 0.0000e+00 Epoch 85/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4483 - auc: 0.8594 - val_loss: 0.3582 - val_auc: 0.0000e+00 Epoch 86/100 1020/1020 [==============================] - 1s 994us/step - loss: 0.4508 - auc: 0.8577 - val_loss: 0.4102 - val_auc: 0.0000e+00 Epoch 87/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4490 - auc: 0.8586 - val_loss: 0.3766 - val_auc: 0.0000e+00 Epoch 88/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4530 - auc: 0.8560 - val_loss: 0.3859 - val_auc: 0.0000e+00 Epoch 89/100 1020/1020 [==============================] - 1s 876us/step - loss: 0.4474 - auc: 0.8599 - val_loss: 0.3496 - val_auc: 0.0000e+00 Epoch 90/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4498 - auc: 0.8579 - val_loss: 0.3402 - val_auc: 0.0000e+00 Epoch 91/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4513 - auc: 0.8573 - val_loss: 0.3869 - val_auc: 0.0000e+00 Epoch 92/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4523 - auc: 0.8562 - val_loss: 0.3777 - val_auc: 0.0000e+00 Epoch 93/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4506 - auc: 0.8579 - val_loss: 0.3949 - val_auc: 0.0000e+00 Epoch 94/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4506 - auc: 0.8577 - val_loss: 0.3546 - val_auc: 0.0000e+00 Epoch 95/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4506 - auc: 0.8575 - val_loss: 0.4198 - val_auc: 0.0000e+00 Epoch 96/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4524 - auc: 0.8566 - val_loss: 0.3546 - val_auc: 0.0000e+00 Epoch 97/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4504 - auc: 0.8582 - val_loss: 0.3942 - val_auc: 0.0000e+00 Epoch 98/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4485 - auc: 0.8593 - val_loss: 0.3837 - val_auc: 0.0000e+00 Epoch 99/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4521 - auc: 0.8565 - val_loss: 0.3795 - val_auc: 0.0000e+00 Epoch 100/100 1020/1020 [==============================] - 1s 1ms/step - loss: 0.4497 - auc: 0.8579 - val_loss: 0.3999 - val_auc: 0.0000e+00
# Capturing learning history per epoch
hist = pd.DataFrame(history.history)
hist['epoch'] = history.epoch
# Plotting accuracy at different epochs
plt.plot(hist['loss'])
plt.plot(hist['val_loss'])
plt.legend(("train" , "valid") , loc =0)
<matplotlib.legend.Legend at 0x296269aa8e0>
score = model.evaluate(X_test, y_test)
63/63 [==============================] - 0s 1ms/step - loss: 0.5484 - auc: 0.8513
print(score)
[0.5483859181404114, 0.851285994052887]
## Confusion Matrix on unsee test set
import seaborn as sn
y_pred1 = model.predict(X_test)
for i in range(len(y_test)):
if y_pred1[i]>0.5:
y_pred1[i]=1
else:
y_pred1[i]=0
cm2=confusion_matrix(y_test, y_pred1)
labels = ['True Negative','False Positive','False Negative','True Positive']
categories = [ 'Non_Exited','Exited']
make_confusion_matrix(cm2,
group_names=labels,
categories=categories,
cmap='Blues')
When focusing on recall as the target metric the Decision Tree model gives us better results as compared to the Neural Network model which has really poor performance in terms of false positives. However, when focusing on AUC-ROC as in the second iteration of the Neural Network, we actually end up with a much more balanced model. The final model has not only improved over the decision tree from the AUC-ROC stat, but recall as well.